The invention relates to a vehicle assistance system. The invention also relates to a corresponding computer implemented method and computer program product.
Recent advances in assisting a driver operating a vehicle have successfully emerged into the market as they compensate shortcomings of human drivers, such as inevitable reaction times for emergency brakes or deficiencies for vehicle stabilization. Systems providing such a drive assistance may typically be installed with the vehicle and comprise a suitable sensor arrangement (e.g. including radar, lidar, cameras) for detecting the vehicle environment and determine possible collision objects. Such a system may additionally be connected to the break arrangement of the vehicle and thus be capable of completely preventing an imminent collision or at least minimizing the consequences of a collision by an autonomously initiated full brake application.
Typical implementations of the above discussed functionality may for example find its way into automatic cruise control systems (ACC system) or in relation to (full or semi) autonomous vehicles, where e.g. the ACC system allows adaptability in regards to a preceding second vehicle such that the vehicle-to-vehicle distance is kept long enough to minimize a collision. In some instances, the ACC system may also take into account a further (e.g. third) vehicle placing itself in between the own vehicle and the second vehicle, possibly making it necessary to (emergency) break the own vehicle.
Even though the above discussed implementations for driver assistance minimize the collision risk, they are generally based on obstacles being direct and readily visible when seen from a perspective of the vehicle or driver of the vehicle. Accordingly, it could be interesting to be able to acquire information collected by further sensor arrangements positioned in the surrounding of the own vehicle for allowing assessment of a possible upcoming traffic situation, i.e. where the information collected by the further sensor arrangements holds information not readily visible from the own vehicle.
According to an aspect of the invention, a vehicle assistance system for a first vehicle is provided, comprising communication circuitry provided for image communication, and a control unit connected to the communication circuitry, wherein the control unit is configured to identify, using the communication circuitry, at least one external camera located remotely away from and within a predefined perimeter surrounding the first vehicle, acquire, using the communication circuitry, an image feed produced by the at least one external camera, determine at least one present or upcoming vehicle scenario for the first vehicle based on vehicle information extracted from an internal source of the first vehicle, the vehicle scenario including information relating to a current location of the first vehicle, determine a safety requirement for the vehicle scenario, correlate the image feed produced by the at least one external camera and the safety requirement for the vehicle scenario, and operate the first vehicle based on the image feed if the correlation between the image feed and the safety requirement indicates a match.
In accordance to the invention, the communication circuitry is configured to automatically identify if any external cameras are located within the surrounding of the own vehicle (the first vehicle) and if any such cameras are available, acquire an image feed from the cameras. Based on the operation of the vehicle, a safety requirement is determined, where the safety requirement is correlated with the image feed from the external camera(s) to see if there is a match. In case there is a match, actions may be taken for the own vehicle based on the content of the image feed.
Accordingly, visual information not readily available when seen from the perspective of the own vehicle may be used for improving the operational safety for the vehicle. That is, the image feed acquired from the external camera(s) typically holds visual illustrations of situations/scenarios that currently may not be seen from the own vehicle, and this information may be used for giving an improved understanding of the surrounding, such that suitable decisions for upcoming traffic scenarios may be taken even before they are taking place.
The expression “operate the first vehicle based on the image feed” should be interpreted in the broadest sense. For example, in a possible implementation of the invention, the vehicle may be an autonomous vehicle (full or semi), where the control unit may be configured for extracting specific features from the acquired image feed. The extracted features could subsequently be used for controlling for example a speed, a lane selection, a gear selection, etc. for the own/first vehicle. In another possible implementation of the invention, the vehicle may be a user operated vehicle. In such an implementation the operation of the vehicle may take a somewhat different approach, such as for example by providing at least a portion of the image feed to a display comprised with the vehicle assistance system, for example arranged in a driver's compartment of the vehicle. Accordingly, in such an implementation the image feed may be seen as an “extended view” that may be useful for the driver when operating the vehicle.
In accordance to the invention, the internal source of the first vehicle to be used for determining at least one present or upcoming vehicle scenario should be understood to include any type of component or element comprised with the first vehicle that may give information in regards to the operation of the vehicle. For example, the internal source could in one embodiment be a sensor comprised with the first vehicle. Such a sensor could for example be a sensor for determining a steering angle of the vehicle, a speed sensor, an inclination sensor, an acceleration sensor, etc. The internal source could also, in combination and/or alternatively, be part of a user interface comprised with the vehicle, such as for example a gear leaver, a turn signal leaver, etc. The internal source could also, within the scope of the invention, be a so called “virtual sensor”, where collaborative vehicle environmental data is used for creating an output, similar to what could be expected from a “normal” sensor. For example, a yaw rate of the vehicle could be determined using e.g. GPS parameters, generating an output corresponding to what could be expected from a yaw rate sensor.
Typically the different internal vehicle sources as mentioned above may be switched between a plurality of predetermined states (on/off, between different gears, etc.), where the different states may be used as an indication of a present and upcoming vehicle scenario. For example, in case the driver of the vehicle shifts to a reverse gear, this information may be used for determining that the vehicle scenario is a reversal of the vehicle. The information from the internal vehicle sources may be distributed within an internal network of the vehicle, such as for example a CAN bus comprised with the vehicle.
In an embodiment of the invention, the vehicle scenario is selected from a predetermined set of vehicle scenarios, and at least one safety requirement is predetermined for each vehicle scenario of the set of vehicle scenarios. As understood from the above, the expression “vehicle scenario” should be understood to mean any type of operation and/or context that the vehicle may be in, at present or e.g. the near future. In relation to the above example, the vehicle scenario may be determined to be “reversal of the vehicle” based on the operation of the gear leaver. The vehicle scenario will in turn have a related safety requirement, for example predefined as being “a backward view where an obstacle is detectable”. Hence, in case the vehicle is to be reversed (vehicle scenario), then the vehicle assistance system tries to identify an external camera that provided an image feed that will provide assistance during operation of the vehicle such that a potential obstacle behind the vehicle may be identified (safety requirement).
In accordance to the invention, the external camera is to be located within a predefined perimeter surrounding the (first) vehicle, however not comprised with the (first) vehicle. The expression “predefined perimeter” should be interpreted broadly. It could be possible to for example define the “size” of the perimeter based on the vehicle scenario. Thus, in travelling forward at a “highway speed”, the perimeter may be made large as compared to a scenario where the vehicle is reversing at an in comparison low speed (high maneuverability). The perimeter may not necessarily be completely surrounding the vehicle. For example when based on the vehicle scenario and when reversing, the perimeter may be defined to be solely backward of the vehicle. Similar scenarios of course exist when travelling in the forward direction.
Preferably, the correlation of the image feed produced by the at least one external camera and the safety requirement for the vehicle scenario is further based on a coverage area for the at least one external camera. The expression “coverage area” should be understood to include for example a view angle and a useful distance range for the camera, as well as a coverage area defined by e.g. a first, a second and a third coordinate.
The location of the vehicle is in accordance to the invention and related to the concept of “perimeter” for the external camera. The location of the first vehicle could for example be acquired from a receiver for a Global Navigation Satellite System (GNSS, e.g. GPS, etc.) comprised with first vehicle. The vehicle information may also be location based, e.g. comprising map data acquired from a storage device comprised with the vehicle. The map data may be permanently of intermediately stored in the vehicle. When stored in an intermediate fashion, the map data may be acquired “on the go” from a remote/cloud server.
In a preferred embodiment of the invention the first vehicle further comprises a vehicle camera connected to the control unit, and the control unit is further configured to combining an image stream produced by the camera of the first vehicle with the image stream produced by the at least one external camera, and display the combined image stream using the display screen. Accordingly, it may in accordance to the invention be possible to e.g. “stich” image data acquired from the vehicle camera with the image stream acquired from the (at least one) external camera. The combination of image data is typically based on the location of the external camera, the coverage area of the external camera, the (current) position of the vehicle camera, and a corresponding coverage area for the vehicle camera.
In a possible embodiment of the invention, the control unit is further configured to acquire external information relating to the surrounding of the vehicle, and determine the at least one present or upcoming vehicle scenario based on the external information. Such information may for example be acquired from “infrastructure components” in the surrounding of the vehicle, from the external camera, and/or from a second/further/other vehicle within the predefined perimeter of the first vehicle. Each of the mentioned “external components” will in such an implementation include communication capability and will be able to transmit “relevant” information to the first vehicle. Communication between the first and the second vehicle for providing instructions and/or warnings may be in any form, typically based on radio or optical communication (e.g. vehicle-to-vehicle communication). Similarly, communication between a fixed positioned external camera, an infrastructure component (e.g. transmitting weather information, information relating to traffic congestions, etc.), etc. may be provided in a similar manner, for example defined as infrastructure-to-vehicle communication.
According to another aspect of the present invention there is provided a camera arrangement configured for use with a vehicle assistance system, wherein the camera arrangement comprises an image sensor configured to capture an image stream, a communication module configured for image communication, and an image processing circuitry connected to the communication module and the image sensor, wherein the image processing circuitry is configured to receive a request from an external communication node for the image stream, the request comprises a location of the external communication node, comparing a location of the camera arrangement and the location of the external communication node, and transmit the image stream to the external communication node if the external communication node is within a predefined perimeter surrounding the camera arrangement.
The camera arrangement is typically configured for use with the vehicle assistance system as discussed above, providing similar advantages.
According to a further aspect of the present invention there is provided a computer implemented method for operating a vehicle assistance system for a first vehicle, the vehicle assistance system comprising communication circuitry provided for image communication, and a control unit connected to the communication circuitry, wherein the method comprises the steps of identifying at least one external camera located remotely away from and within a predefined perimeter surrounding the first vehicle, acquiring an image feed produced by at least one external camera, determining at least one present or upcoming vehicle scenario for the first vehicle based on vehicle information extracted from an internal source of the first vehicle, the vehicle scenario including information relating to a current location of the first vehicle, determining a safety requirement for the vehicle scenario, correlating the image feed produced by the at least one external camera and the safety requirement for the vehicle scenario, and operating the first vehicle based on the image feed if the correlation between the image feed and the safety requirement indicates a match.
According to a still further aspect of the present invention there is provided a computer program product comprising a computer readable medium having stored thereon computer program means for operating a vehicle assistance system for a first vehicle, the vehicle assistance system comprising communication circuitry provided for image communication, and a control unit connected to the communication circuitry, the computer program product comprising code for identifying at least one external camera located remotely away from and within a predefined perimeter surrounding the first vehicle, code for acquiring an image feed produced by at least one external camera, code for determining at least one present or upcoming vehicle scenario for the first vehicle based on vehicle information extracted from an internal source of the first vehicle, the vehicle scenario including information relating to a current location of the first vehicle, code for determining a safety requirement for the vehicle scenario, code for correlating the image feed produced by the at least one external camera and the safety requirement for the vehicle scenario, and code for operating the first vehicle based on the image feed if the correlation between the image feed and the safety requirement indicates a match.
The computer readable medium may be any type of memory device, including one of a removable nonvolatile random access memory, a hard disk drive, a floppy disk, a CD-ROM, a DVD-ROM, a USB memory, an SD memory card, or a similar computer readable medium known in the art.
Further advantages and advantageous features of the invention are disclosed in the following description and in the dependent claims.